Applying Machine Translation Evaluation Techniques to Textual CBR
نویسندگان
چکیده
The need for automated text evaluation is common to several AI disciplines. In this work, we explore the use of Machine Translation (MT) evaluation metrics for Textual Case Based Reasoning (TCBR). MT and TCBR typically propose textual solutions and both rely on human reference texts for evaluation purposes. Current TCBR evaluation metrics such as precision and recall employ a single human reference but these metrics are misleading when semantically similar texts are expressed with different sets of keywords. MT metrics overcome this challenge with the use of multiple human references. Here, we explore the use of multiple references as opposed to a single reference applied to incident reports from the medical domain. These references are created introspectively from the original dataset using the CBR similarity assumption. Results indicate that TCBR systems evaluated with these new metrics are closer to human judgements. The generated text in TCBR is typically similar in length to the reference since it is a revised form of an actual solution to a similar problem, unlike MT where generated texts can sometimes be significantly shorter. We therefore discovered that some parameters in the MT evaluation measures are not useful for TCBR due to the intrinsic difference in the text generation process.
منابع مشابه
Using T-Ret System to Improve Incident Report Retrieval
This papers describes novel research involving the development of Textual CBR techniques and applying them to the problem of Incident Report Retrieval. Incident Report Retrieval is a relatively new research area in the domain of Accident Reporting and Analysis. We describe T-Ret, an Incident Report Retrieval system that incorporates textual CBR techniques and outline preliminary evaluation resu...
متن کاملDeveloping Mapping and Evaluation Techniques for Textual Case-Based Reasoning
Textual Case-Based Reasoning (CBR) is not simply Information Retrieval (IR) of text documents which happen also to be cases. Nor does it involve only techniques for automatically determining what cases represented as texts are about or techniques for automatically indexing such cases under relevant features. Textual CBR is still case-based reasoning, and for us, that means drawing inferences ab...
متن کاملEvaluation of Textual CBR Approaches
Evaluation is a crucial step in a research project, it demonstrates how well the chosen approach and the implemented techniques work, and can uncover limitations as well as point toward improvements and future research. A formal evaluation also facilitates comparing the project to previous work, and enables other researchers to assess its usefulness to their problems. Evaluating Textual CBR sys...
متن کاملUoW: NLP techniques developed at the University of Wolverhampton for Semantic Similarity and Textual Entailment
This paper presents the system submitted by University of Wolverhampton for SemEval-2014 task 1. We proposed a machine learning approach which is based on features extracted using Typed Dependencies, Paraphrasing, Machine Translation evaluation metrics, Quality Estimation metrics and Corpus Pattern Analysis. Our system performed satisfactorily and obtained 0.711 Pearson correlation for the sema...
متن کاملSemantic Textual Similarity for MT evaluation
This paper describes the system used for our participation in the WMT12 Machine Translation evaluation shared task. We also present a new approach to Machine Translation evaluation based on the recently defined task Semantic Textual Similarity. This problem is addressed using a textual entailment engine entirely based on WordNet semantic features. We described results for the Spanish-English, C...
متن کامل